Comparing Possibly Misspecied Forecasts

Recent work has emphasized the importance of evaluating forecasts of a statistical functional (such as a mean, quantile, or distribution) using a loss function that is consistent for the functional of interest, of which there are an in…nite number. If forecasters all use correctly speci…ed models free from estimation error, and if the information sets of competing forecasters are nested, then the ranking induced by a single consistent loss function is su¢ cent for the ranking by any consistent loss function. This paper shows, via analytical results and realistic simulation-based analyses, that the presence of misspeci…ed models, parameter estimation error, or nonnested information sets, leads generally to sensitivity to the choice of (consistent) loss function. Thus, rather than merely specifying the target functional, which narrows the set of relevant loss functions only to the class of loss functions consistent for that functional, forecast consumers or survey designers should specify the single speci…c loss function that will be used to evaluate forecasts. An application to survey forecasts of US in‡ation illustrates the results. Keywords: Survey forecasts, economic forecasting, point forecasting, model misspeci…cation, Bregman distance, proper scoring rules, consistent loss functions. J.E.L. codes: C53, C52, E37. AMS 2010 Classi…cations: 62M20, 62P20. For helpful comments I thank Tim Bollerslev, Dean Croushore, Frank Diebold, Tilmann Gneiting, Jia Li, Robert Lieli, Minchul Shin, Allan Timmermann and seminar participants at Boston College, Columbia, Duke, Penn, Princeton, St. Louis Federal Reserve, 8 French Economics conference, NBER Summer Institute, Nordic Econometric Society meetings, and the World Congress of the Econometric Society. Contact address: Andrew Patton, Department of Economics, Duke University, 213 Social Sciences Building, Box 90097, Durham NC 27708-0097. Email: [email protected].